Method and system for synchronising social messages with a content timeline

ABSTRACT

This disclosure describes a mechanism for associating a message to a specific point in time relative to the start of media content (such as video). Social messages employing these techniques can be associated to specific points in time within the particular content. For example, social messages can be associated with an opening scene that sparks social commentary and later for other arbitrary points in the media timeline. This example becomes even more relevant when a piece of media is replayed at a later time.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 61/584,134 filed Jan. 6, 2012 which is incorporated by referenceherein in its entirety.

This application is also related to the applications entitled: “METHODSAND SYSTEMS FOR SYNCHRONIZING CONTENT ON A SECOND SCREEN”, “METHOD ANDSYSTEM FOR PROVIDING A GRAPHICAL REPRESENTATION ON A SECOND SCREEN OFSOCIAL MESSAGES RELATED TO CONTENT ON A FIRST SCREEN”, “METHOD ANDSYSTEM FOR PROVIDING A DISPLAY OF SOCIAL MESSAGES ON A SECOND SCREENWHICH IS SYNCHED WITH CONTENT ON A FIRST SCREEN”, “ALTERNATE VIEWPLAYBACK ON A SECOND SCREEN”, and “METHOD AND SYSTEM FOR PROVIDINGDYNAMIC ADVERTISING ON SECOND SCREEN BASED ON SOCIAL MESSAGES” whichhave been filed concurrently and are incorporated by reference herein intheir entirety.

BACKGROUND

1. Technical Field

The present invention generally relates to providing additional contentrelated to displayed content.

2. Description of Related Art

Social messages are typically ephemeral and relevant only to the presentmoment in time. Many social messages are associated with a particularevent or topic. One common convention for associating a message to aparticular event or topic is to use hashtags. These hashtags provide ameans to filter for social messages of interest. Filtering for thesehashtags provides a “real-time” view of messages on the topic. Usingsuch association conventions social messages can be understood to referto content being viewed. However, such associations, such as hashtags,do not provide a specific indication of what period of time in thecontent the message is referencing.

SUMMARY

This disclosure describes a mechanism for associating a message to aspecific point in time relative to the start of media content (such asvideo). Social messages employing these techniques can be associated tospecific points in time within the particular content. For example,social messages can be associated with an opening scene that sparkssocial commentary and later for other arbitrary points in the mediatimeline. This example becomes even more relevant when a piece of mediais replayed at a later time.

In accordance with one embodiment, a method for synching social messagesto a content timeline is provided. The method includes the steps ofmonitoring social media for a message relating to displayed content,processing the message to associate the messages with a period of timein the display of the content, and associating a timestamp with themessage indicating the period of time in the display of the content forwhich the message is associated.

In accordance with another embodiment, a method for processing socialmedia synched with a content timeline is provided. The method includesthe steps of receiving a message relating to displayed content,determining if a timestamp is associated with message indicating theperiod of time in the display of the content for which the message isassociated, and providing the message synched to the display of thecontent.

In accordance with another embodiment, a second screen device capable ofprocessing social media synched with a content timeline is provided. Thesecond screen device includes a screen, storage, and a processor. Thescreen is configured to display content. The storage is for storingdata. The processor is configured to a processor configured to receive amessage relating to displayed content, determine if a timestamp isassociated with message indicating the period of time in the display ofthe content for which the message is associated, and provide the messagesynched to the display of the content

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system diagram outlining the delivery of video and audiocontent to the home in accordance with one embodiment.

FIG. 2 is system diagram showing further detail of a representative settop box receiver.

FIG. 3 is a diagram depicting a touch panel control device in accordancewith one embodiment.

FIG. 4 is a diagram depicting some exemplary user interactions for usewith a touch panel control device in accordance with one embodiment.

FIG. 5 is system diagram depicting one embodiment of a system forimplementing techniques of the present invention in accordance with oneembodiment.

FIG. 6 is a flow diagram depicting an exemplary process in accordancewith one embodiment.

FIG. 7 is a diagram depicting an exemplary methodology of synchingbetween devices in accordance with one embodiment.

FIG. 8 is a diagram depicting an exemplary methodology of synchingbetween devices in accordance with one embodiment.

FIGS. 9A-9F are exemplary skeletal screen views depicting features inaccordance with one embodiment when used in passive mode.

FIGS. 10A-10D are exemplary skeletal screen views depicting features inaccordance with one embodiment when used in active mode.

FIGS. 11A-11C are exemplary skeletal views depicting a social mediasharing feature in accordance with one embodiment.

FIGS. 12A and 12B are exemplary skeletal views depicting a contentselection features in accordance with one embodiment.

FIGS. 13A-13E are exemplary skeletal views depicting additional featuresin accordance with one embodiment.

FIGS. 14A-14L are exemplary skinned screen views depicting how certainfeatures could appear to a user.

FIG. 15 is a exemplary skeletal view depicting social media features inaccordance with one embodiment.

FIG. 16 is a flow diagram depicting the functionality of social mediafeatures in accordance with one embodiment.

FIG. 17 is an exemplary skinned screen view depicting social mediafeatures in accordance with one embodiment.

FIG. 18 is a flow diagram providing a general methodology of providingcontent on a second screen based on social messages regarding contentbeing displayed on a first screen in accordance with one embodiment.

FIG. 19 is an exemplary view of a social quote event in accordance withone embodiment.

FIG. 20 is an exemplary view of dynamic advertising on a second screenbased on social messages in accordance with one embodiment.

FIG. 21 is an exemplary view of how a social message can be associatedwith content in accordance with one embodiment.

FIG. 22 is flow diagram providing a methodology for generating a socialmessage associated with content in accordance with one embodiment.

FIG. 23 is flow diagram providing a methodology for processing a socialmessage associated with content in accordance with one embodiment.

FIG. 24 is flow diagram providing a methodology for providing agraphical representation of social messages associated with time periodsof content in accordance with one embodiment.

FIG. 25 is an exemplary representation of bins that make up a graphicalrepresentation of social messages associated with time periods ofcontent in accordance with one embodiment.

FIG. 26 is an exemplary representation of the association of messages tothe bins that make up a graphical representation of social messagesassociated with time periods of content in accordance with oneembodiment.

FIG. 27 is a exemplary skeletal view depicting a graphicalrepresentation of social messages associated with time periods ofcontent in accordance with one embodiment.

FIG. 28 is an exemplary view of the operation of a graphicalrepresentation of social messages associated with time periods ofcontent in accordance with one embodiment.

FIG. 29 is another exemplary view of the operation of a graphicalrepresentation of social messages associated with time periods ofcontent in accordance with another embodiment.

FIG. 30 is another exemplary view of the operation of a graphicalrepresentation of social messages associated with time periods ofcontent in accordance with another embodiment.

FIG. 31 is another exemplary view of the operation of a graphicalrepresentation of social messages associated with time periods ofcontent in accordance with another embodiment.

FIG. 32 is another exemplary view of the operation of a graphicalrepresentation of social messages associated with time periods ofcontent in accordance with another embodiment.

FIG. 33 is a exemplary skinned screen view depicting a graphicalrepresentation of social messages associated with time periods ofcontent in accordance with one embodiment.

DETAILED DESCRIPTION

Turning now to FIG. 1, a block diagram of an embodiment of a system 100for delivering content to a home or end user is shown. The contentoriginates from a content source 102, such as a movie studio orproduction house. The content may be supplied in at least one of twoforms. One form may be a broadcast form of content. The broadcastcontent is provided to the broadcast affiliate manager 104, which istypically a national broadcast service, such as the AmericanBroadcasting Company (ABC), National Broadcasting Company (NBC),Columbia Broadcasting System (CBS), etc. The broadcast affiliate managermay collect and store the content, and may schedule delivery of thecontent over a deliver network, shown as delivery network 1 (106).Delivery network 1 (106) may include satellite link transmission from anational center to one or more regional or local centers. Deliverynetwork 1 (106) may also include local content delivery using localdelivery systems such as over the air broadcast, satellite broadcast, orcable broadcast. The locally delivered content is provided to areceiving device 108 in a user's home, where the content willsubsequently be searched by the user. It is to be appreciated that thereceiving device 108 can take many forms and may be embodied as a settop box/digital video recorder (DVR), a gateway, a modem, etc. Further,the receiving device 108 may act as entry point, or gateway, for a homenetwork system that includes additional devices configured as eitherclient or peer devices in the home network.

A second form of content is referred to as special or additionalcontent. Special or additional content may include content delivered aspremium viewing, pay-per-view, or other content otherwise not providedto the broadcast affiliate manager, e.g., movies, video games or othervideo elements. In many cases, the special content may be contentrequested by the user. The special content may be delivered to a contentmanager 110. The content manager 110 may be a service provider, such asan Internet website, affiliated, for instance, with a content provider,broadcast service, or delivery network service. The content manager 110may also incorporate Internet content into the delivery system. Thecontent manager 110 may deliver the content to the user's receivingdevice 108 over a separate delivery network, delivery network 2 (112).Delivery network 2 (112) may include high-speed broadband Internet typecommunications systems. It is important to note that the content fromthe broadcast affiliate manager 104 may also be delivered using all orparts of delivery network 2 (112) and content from the content manager110 may be delivered using all or parts of delivery network 1 (106). Inaddition, the user may also obtain content directly from the Internetvia delivery network 2 (112) without necessarily having the contentmanaged by the content manager 110.

Several adaptations for utilizing the separately delivered additionalcontent may be possible. In one possible approach, the additionalcontent is provided as an augmentation to the broadcast content,providing alternative displays, purchase and merchandising options,enhancement material, etc. In another embodiment, the additional contentmay completely replace some programming content provided as broadcastcontent. Finally, the additional content may be completely separate fromthe broadcast content, and may simply be a media alternative that theuser may choose to utilize. For instance, the additional content may bea library of movies that are not yet available as broadcast content.

The receiving device 108 may receive different types of content from oneor both of delivery network 1 and delivery network 2. The receivingdevice 108 processes the content, and provides a separation of thecontent based on user preferences and commands. The receiving device 108may also include a storage device, such as a hard drive or optical diskdrive, for recording and playing back audio and video content. Furtherdetails of the operation of the receiving device 108 and featuresassociated with playing back stored content will be described below inrelation to FIG. 2. The processed content is provided to a displaydevice 114. The display device 114 may be a conventional 2-D typedisplay or may alternatively be an advanced 3-D display.

The receiving device 108 may also be interfaced to a second screen suchas a touch screen control device 116. The touch screen control device116 may be adapted to provide user control for the receiving device 108and/or the display device 114. The touch screen device 116 may also becapable of displaying video content. The video content may be graphicsentries, such as user interface entries, or may be a portion of thevideo content that is delivered to the display device 114. The touchscreen control device 116 may interface to receiving device 108 usingany well known signal transmission system, such as infra-red (IR) orradio frequency (RF) communications and may include standard protocolssuch as infra-red data association (IRDA) standard, Wi-Fi, Bluetooth andthe like, or any other proprietary protocols. In some embodiments, thetouch screen control device 116 can be interfaced directly with deliverynetworks 1 and 2. Operations of touch screen control device 116 will bedescribed in further detail below.

In the example of FIG. 1, the system 100 also includes a back end server118 and a usage database 120. The back end server 118 includes apersonalization engine that analyzes the usage habits of a user andmakes recommendations based on those usage habits. The usage database120 is where the usage habits for a user are stored. In some cases, theusage database 120 may be part of the back end server 118 a. In thepresent example, the back end server 118 (as well as the usage database120) is connected to the system the system 100 and accessed through thedelivery network 2 (112).

Turning now to FIG. 2, a block diagram of an embodiment of a receivingdevice 200 is shown. Receiving device 200 may operate similar to thereceiving device described in FIG. 1 and may be included as part of agateway device, modem, set-top box, or other similar communicationsdevice. The device 200 shown may also be incorporated into other systemsincluding an audio device or a display device. In either case, severalcomponents necessary for complete operation of the system are not shownin the interest of conciseness, as they are well known to those skilledin the art.

In the device 200 shown in FIG. 2, the content is received by an inputsignal receiver 202. The input signal receiver 202 may be one of severalknown receiver circuits used for receiving, demodulation, and decodingsignals provided over one of the several possible networks includingover the air, cable, satellite, Ethernet, fiber and phone line networks.The desired input signal may be selected and retrieved by the inputsignal receiver 202 based on user input provided through a controlinterface or touch panel interface 222. Touch panel interface 222 mayinclude an interface for a touch screen device. Touch panel interface222 may also be adapted to interface to a cellular phone, a tablet, amouse, a high end remote or the like.

The decoded output signal is provided to an input stream processor 204.The input stream processor 204 performs the final signal selection andprocessing, and includes separation of video content from audio contentfor the content stream. The audio content is provided to an audioprocessor 206 for conversion from the received format, such ascompressed digital signal, to an analog waveform signal. The analogwaveform signal is provided to an audio interface 208 and further to thedisplay device or audio amplifier. Alternatively, the audio interface208 may provide a digital signal to an audio output device or displaydevice using a High-Definition Multimedia Interface (HDMI) cable oralternate audio interface such as via a Sony/Philips DigitalInterconnect Format (SPDIF). The audio interface may also includeamplifiers for driving one more sets of speakers. The audio processor206 also performs any necessary conversion for the storage of the audiosignals.

The video output from the input stream processor 204 is provided to avideo processor 210. The video signal may be one of several formats. Thevideo processor 210 provides, as necessary a conversion of the videocontent, based on the input signal format. The video processor 210 alsoperforms any necessary conversion for the storage of the video signals.

A storage device 212 stores audio and video content received at theinput. The storage device 212 allows later retrieval and playback of thecontent under the control of a controller 214 and also based oncommands, e.g., navigation instructions such as fast-forward (FF) andrewind (Rew), received from a user interface 216 and/or touch panelinterface 222. The storage device 212 may be a hard disk drive, one ormore large capacity integrated electronic memories, such as static RAM(SRAM), or dynamic RAM (DRAM), or may be an interchangeable optical diskstorage system such as a compact disk (CD) drive or digital video disk(DVD) drive.

The converted video signal, from the video processor 210, eitheroriginating from the input or from the storage device 212, is providedto the display interface 218. The display interface 218 further providesthe display signal to a display device of the type described above. Thedisplay interface 218 may be an analog signal interface such asred-green-blue (RGB) or may be a digital interface such as HDMI. It isto be appreciated that the display interface 218 will generate thevarious screens for presenting the search results in a three dimensionalgrid as will be described in more detail below.

The controller 214 is interconnected via a bus to several of thecomponents of the device 200, including the input stream processor 202,audio processor 206, video processor 210, storage device 212, and a userinterface 216. The controller 214 manages the conversion process forconverting the input stream signal into a signal for storage on thestorage device or for display. The controller 214 also manages theretrieval and playback of stored content. Furthermore, as will bedescribed below, the controller 214 performs searching of content andthe creation and adjusting of the grid display representing the content,either stored or to be delivered via the delivery networks, describedabove.

The controller 214 is further coupled to control memory 220 (e.g.,volatile or nonvolatile memory, including RAM, SRAM, DRAM, ROM,programmable ROM (PROM), flash memory, electronically programmable ROM(EPROM), electronically erasable programmable ROM (EEPROM), etc.) forstoring information and instruction code for controller 214. Controlmemory 220 may store instructions for controller 214. Control memory mayalso store a database of elements, such as graphic elements containingcontent. The database may be stored as a pattern of graphic elements.Alternatively, the memory may store the graphic elements in identifiedor grouped memory locations and use an access or location table toidentify the memory locations for the various portions of informationrelated to the graphic elements. Additional details related to thestorage of the graphic elements will be described below. Further, theimplementation of the control memory 220 may include several possibleembodiments, such as a single memory device or, alternatively, more thanone memory circuit communicatively connected or coupled together to forma shared or common memory. Still further, the memory may be includedwith other circuitry, such as portions of bus communications circuitry,in a larger circuit.

The user interface process of the present disclosure employs an inputdevice that can be used to express functions, such as fast forward,rewind, etc. To allow for this, a touch panel device 300 may beinterfaced via the user interface 216 and/or touch panel interface 222of the receiving device 200, as shown in FIG. 3. The touch panel device300 allows operation of the receiving device or set top box based onhand movements, or gestures, and actions translated through the panelinto commands for the set top box or other control device. In oneembodiment, the touch panel 300 may simply serve as a navigational toolto navigate the grid display. In other embodiments, the touch panel 300will additionally serve as the display device allowing the user to moredirectly interact with the navigation through the grid display ofcontent. The touch panel device may be included as part of a remotecontrol device containing more conventional control functions such asactuator or activator buttons. The touch panel 300 can also includes atleast one camera element. In some embodiments, the touch panel 300 mayalso include a microphone.

Turning now to FIG. 4, the use of a gesture sensing controller or touchscreen, such as shown, provides for a number of types of userinteraction. The inputs from the controller are used to define gesturesand the gestures, in turn, define specific contextual commands. Theconfiguration of the sensors may permit defining movement of a user'sfingers on a touch screen or may even permit defining the movement ofthe controller itself in either one dimension or two dimensions.Two-dimensional motion, such as a diagonal, and a combination of yaw,pitch and roll can be used to define any three-dimensional motion, suchas a swing. A number of gestures are illustrated in FIG. 4. Gestures areinterpreted in context and are identified by defined movements made bythe user.

Bumping 420 is defined by a two-stroke drawing indicating pointing inone direction, either up, down, left or right. The bumping gesture isassociated with specific commands in context. For example, in aTimeShifting mode, a left-bump gesture 420 indicates rewinding, and aright-bump gesture indicates fast-forwarding. In other contexts, a bumpgesture 420 is interpreted to increment a particular value in thedirection designated by the bump. Checking 440 is defined as in drawinga checkmark. It is similar to a downward bump gesture 420. Checking isidentified in context to designate a reminder, user tag or to select anitem or element. Circling 440 is defined as drawing a circle in eitherdirection. It is possible that both directions could be distinguished.However, to avoid confusion, a circle is identified as a single commandregardless of direction. Dragging 450 is defined as an angular movementof the controller (a change in pitch and/or yaw) while pressing a button(virtual or physical) on the tablet 300 (i.e., a “trigger drag”). Thedragging gesture 450 may be used for navigation, speed, distance,time-shifting, rewinding, and forwarding. Dragging 450 can be used tomove a cursor, a virtual cursor, or a change of state, such ashighlighting outlining or selecting on the display. Dragging 450 can bein any direction and is generally used to navigate in two dimensions.However, in certain interfaces, it is preferred to modify the responseto the dragging command. For example, in some interfaces, operation inone dimension or direction is favored with respect to other dimensionsor directions depending upon the position of the virtual cursor or thedirection of movement. Nodding 460 is defined by two fast trigger-dragup-and-down vertical movements. Nodding 460 is used to indicate “Yes” or“Accept.” X-ing 470 is defined as in drawing the letter “X.” X-ing 470is used for “Delete” or “Block” commands. Wagging 480 is defined by twotrigger-drag fast back-and-forth horizontal movements. The wagginggesture 480 is used to indicate “No” or “Cancel.”

Depending on the complexity of the sensor system, only simple onedimensional motions or gestures may be allowed. For instance, a simpleright or left movement on the sensor as shown here may produce a fastforward or rewind function. In addition, multiple sensors could beincluded and placed at different locations on the touch screen. Forinstance, a horizontal sensor for left and right movement may be placedin one spot and used for volume up/down, while a vertical sensor for upand down movement may be place in a different spot and used for channelup/down. In this way specific gesture mappings may be used.

The system and methodology can be implemented in any number of waysdepending on the hardware and the content involved. Examples of suchdeployment include DVD, Blu-Ray disc (BD); streaming video or video ondemand (VOD), and broadcast (satellite, cable, over the air). Each ofthese deployments would have different architectures but one couldstandardize the triggers for each of these events (the additionalcontent) that represents what would be queued by the application runningon the second screen. For example, event A and event B would betriggered by a synching mechanism associated with any of these sourcesof a video. When the tablet encounters “event A”, the program running onthe second screen device (e.g. tablet) will enact “event A”. Similarly,if “event B” is encountered, the program running on the second screendevice would do “event B”.

FIG. 5 depicts a generic system 500 on which such methodology could beimplemented. Here the system 500 includes a first screen device 510, asecond screen device 520, a playback device 530, a network 540 andserver 550. Each of these elements will be discussed in more detailbelow.

The first screen device 510 is a display device, such as display device114 described above in relation to FIG. 1, for displaying content suchas television programs, movies, and websites. Examples of such firstscreen display devices include, but are not limited to, a television,monitor, projector, or the like. The first screen device 510 isconnected to the playback device 530 which can provide the primarycontent to the first screen device 510 for display. Examples of suchcommunication include, but are not limited to HDMI, VGA, Display port,USB, component, composite, radio frequency (RF), and infrared (IR), andthe like. In certain embodiments, the first screen display device 510may be connected to the network 540, in either a wired or wireless(WiFi) manner, providing additional connection to the second screendevice 520 and server 550. In some embodiments, the first display device510 may include the functionality of the playback device 530. In stillother embodiments, the first screen display device 510 may be innon-networked communication 560 with the second screen device 520.Examples of such non-networked communication 560 include, but are notlimited to, RF, IR, Blue-Tooth, and other audio communication techniquesand protocols.

The second screen device 520 is device capable of displaying additionalcontent related to the primary content being displayed on the firstscreen device 510. The second screen device may be a touch screencontrol device 116 or touch screen device 300 as described above.Examples of second screen devices include, but are not limited to, asmart phone, tablet, laptop, personal media player (e.g. ipod), or thelike. The second screen device 520 is in communication with playbackdevice 530 using either network 540, non-networked communication 560, orboth. The second screen device 550 is also in communication with theserver 550 via the network 540 for requesting and receiving additionalcontent related to the primary content being displayed on the firstscreen device 510. In some embodiments, the second screen device 520 maybe in networked or non-networked communication 560 with the first screendevice 510. Examples of such non-networked communication 560 include,but are not limited to, RF, IR, Blue-Tooth (BT), audio communicationtechniques and protocols, or the like.

The playback device 530 is device capable of providing primary contentfor display on the first screen device 510. Examples of such playbackdisplay devices include, but are not limited to, a DVD player, Blue-RayDisc (BD) player, game console, receiver device (cable or satellite),Digital Video Recorder (DVR), streaming device, personal computer, orthe like. The playback device 530 is connected to the first screendevice 510 for providing the primary content to the first screen device510 for display. Examples of such connections include, but are notlimited to HDMI, VGA, Display port, USB, component, composite, radiofrequency (RF), and infrared (IR), and the like. The playback device 530is also connected to the network 540, in either a wired or wireless(WiFi) manner, providing connection to the second screen device 520 andserver 550. In some embodiments, the functionality of the playbackdevice 530 may be included in the first screen display device 510. Instill other embodiments, the playback device 530 may be in non-networkedcommunication 560 with the second screen device 520. Examples of suchnon-networked communication 560 include, but are not limited to, RF, IR,Blue-Tooth (BT), and other audio communication techniques and protocols.

The network 540 can be a wired or wireless communication networkimplemented using Ethernet, MoCA, and wireless protocols or acombination thereof. Examples of such a network include, but are notlimited to, delivery network 1 (106) and delivery network 2 (112)discussed above.

The server 550 is a content server configured to provide additionalcontent to the second screen device 520. In certain embodiments, theserver may also provide the primary content for display on the firstscreen device 510. The service is connected to the network 540 and cancommunicate with any of the devices that are also connected. Examples ofsuch a server include, but are not limited to, content source 102,broadcast affiliate manager 104, content manager 110, and the back endserver described above.

FIG. 6 depicts a flow diagram 600 for a methodology for displayingadditional content related to primary content being viewed is disclosed.The method includes the following steps: Displaying primary content on afirst screen device 510 (step 610). Providing, in association with thedisplay of the primary content on the first screen, a synching mechanismto synch additional content (step 620). Displaying, on a second screendevice 520, additional content related to the primary content on thefirst screen 510 that is synched to the content displayed on the firstscreen device according to the synching mechanism (step 630). In certainembodiments, the method also includes the steps of receiving commandsfrom the second screen device 520 to control the display of primarycontent on the first screen device 510 (step 640) and controlling thedisplay of the primary content on the first screen device 510 based onthe commands received from the second screen device 520 (step 650). Eachof these steps will be described in more detail below.

The step of displaying primary content (step 610), such as a movie ortelevision show, is performed on the first screen device 510. Thisinvolves the primary content being provided to the first screen display510. The primary content can be provided by the playback device 530 orbe received directly from a content provider at the first screen displaydevice 510. The primary content is then shown or otherwise displayed onthe first screen device 510. The display of the primary content alsoincludes the control of the content being displayed. This can includethe traditional playback commands of play, stop, pause, rewind, and fastforward as well as the navigation of on screen menus to select thecontent and other playback options. In certain embodiments, the displayon the first screen device 510 (step 620) further includes displaying anindicator of the type of additional content being displayed on thesecond screen device 520.

The provided synching mechanism (step 620) can be implemented in anumber of ways. In certain embodiments the synching mechanism isperformed by an application running on the second screen device 520, theplayback mechanism 530, the first screen device 510 or any combinationthereof. At its most basic, the second screen device 520 is configured(via an application) to detect synching signals, cues, or other type ofindicators that directs the second screen device 520 to update thedisplay of additional content to coincide with the primary content beingdisplayed on the first screen 510. The synching signals, cues or othertype of indicators, can be provided as part of the primary content orcan be generated by the playback device 530 of first screen device 510(via an application) in accordance with the chapter, scene, time-code,subject matter, or content being displayed. The synching signals, cuesor other type of indicators can be transmitted to the second screendevice 520 using the network, in either a wired or wireless (WiFi)manner, or using non-networked communication 560 such as audio signals.Examples of some of the implementations are given below. Other possibleimplementations will be apparent given the benefit of this disclosure.

The step of displaying the additional content, such as supplementalmaterials, video clips, websites, and the like (step 630) is performedon the second screen device 520. The additional content can be storedlocally on the second screen device 520 or be provided by the server550, playback device 530, or first screen device 510. The display of theadditional content is synched to the primary content being displayed onthe first screen device 510 according to the synching mechanism. Forexample, when the second screen device 520 detects a synching signal,cue or other type of indicator, the second screen device 520 updates thedisplay of the additional content accordingly. In some embodiments, thisfurther involves contacting and requesting the additional content fromthe server 550, playback device 530, or first screen device 510 andsubsequently downloading and displaying the additional content. In someembodiments, the additional content to be displayed can be selected,modified, or omitted based on the user using the system.

In certain embodiments, the display on the second screen device 520(step 630) further includes displaying the status of the display of theprimary content on the first screen device 510 such as whether thedisplay of the primary content on the first screen device 510 has beenpaused. In certain other embodiments, the display on the second screendevice 520 (step 630) further includes displaying the status of thesynch between the additional content on the second screen device 520 andthe primary content on the first screen device 510.

In certain embodiments, the second screen device 520 is capable oftransmitting as well as receiving. The optional steps 640 and 650address this capability. In step 640 commands are received from thesecond screen device 520. Ideally, these commands are received at thedevice controlling the playback of the primary content on the firstscreen device 510. In certain embodiment, the playback device 530 is thedevice receiving the commands. The commands can be sent via the network540 or non-networked communication 560. Once received, the commands cancontrol the display of the primary content (step 650). Examples of suchcontrol include, but are not limited to, play, stop, pause, rewind,fast-forward, as well as chapter, scene, and selection. These commandscan also be used to synch the primary content displayed on the firstscreen device 510 with the additional content being displayed on thesecond screen device 520.

FIG. 7 provides a high level overview of one example of system 700 witha synching mechanism implemented using a non-networked communication560. In this system 700, the non-networked communication synchingmechanism is audio watermarking 710. In this example, audio watermarking710 involves inserting a high-frequency signal, cue, or other indicatorinto the audio signal of the primary content being displayed on thefirst screen device 510. The audio watermark is inaudible to humans butcan be detected by a microphone in the second screen device 520. Whenthe second screen device 520 detects an audio watermark, the displayedadditional content is updated to synch with the primary content beingdisplayed on the first screen device 510 based on the detectedwatermark. The audio watermarks can be incorporated into the primarycontent at the source of the content or inserted locally by the playbackdevice 520 or first screen device 510.

FIG. 8 provides a high-level overview of one example of a system 800with a synching mechanism implemented using the network 540. In thissystem 800 the synching mechanism is wireless communication (WiFi) 810between a playback device 530 (a Blu-Ray Disc player) and the secondscreen device 520 (an iOS device running an application). In the exampleof FIG. 7, the features and protocols of a BD-Live enabled device areused. There are two main components of this protocol: connection andcommunication. Both are described below. For simplicity the secondscreen iOS application will be referred to as the “iPad” and the BD-Liveenabled device will be referred to as the “disc”.

Connection occurs when an iOS enabled device 520 first launches thesecond screen application and attempts to connect to a BD-Live enableddevice 530 on the same Wi-Fi network 540.

-   -   1. Disc is inserted into BD player    -   2. Disc enters UDP ‘listening’ loop    -   3. iPad launches second screen application    -   4. iPad performs UDP broadcast of authentication token    -   5. Disc receives authentication token and authenticates    -   6. Disc retrieves IP from tokens sender (iPad's IP)    -   7. Disc responds to authentication with its IP and PORT    -   8. iPad confirms IP and PORT    -   9. iPad closes UDP socket communication    -   10. iPad establishes direct TCP socket communication with disc        based on IP and PORT provided.

Communication occurs after a connection has been established between thesecond screen iOS application and a BD-Live enabled device.

-   -   1. iPad and Disc are aware of each other's IP's as well as what        PORT communication should occur using    -   2. TCP socket communication is maintained for the duration of        the applications lifecycle.

One advantage of such a wireless communication as seen in this exampleis that it is bi-directional allowing the second screen device totransmit as well as receive commands. This allows for two way synchingas well as control of playback from the second screen device 520.

In certain embodiments, the application of the second screen device 520could be specific to a specific program or movie on a specific system(e.g. BD). In other embodiments, the second screen application could begeneric to a studio with available plug-ins to configure the applicationto a particular program or movie. In still other embodiments the secondscreen application could be universal to system (BD, VOD, broadcast),content, or both. Other possible implementations and configurations willbe apparent to one skilled in the art given the benefit of thisdisclosure.

The system can be operated in with a passive approach or an interactiveapproach. In the passive approach icons displayed on first screen device510 prompt the user to look at the second screen device 520 for anadditional content event being displayed that is related to the primarycontent displayed on the first screen device 510. The icon preferablyindicates what type of additional content event is available on thesecond screen device 520 (e.g., a shopping cart icon indicates apurchase event, an “I” icon indicates an information event, a stickmanicon indicates a character information event, etc.) FIG. 9A-F depictssome of the aspects that may be displayed to the user in passive mode.

FIGS. 9A-F depict skeletal examples of what may be displayed on thescreen 900 of the second screen device to a user when using anapplication in passive mode that provides additional content on thesecond screen device 520 that is synched with the primary content on thefirst screen device 510.

FIG. 9A depicts a splash screen that may be displayed to the user whenthe application is launched. It includes the product logo and indicationof the primary content 902. Here new content screens transition in fromright in a conveyer-belt like manner as indicate by arrow 904.

FIG. 9B depicts a pop-up message 906 that is displayed to a user when noplayback device 530 is detected by second screen device 520.

The screen 900 of FIG. 9C shows a synch button/icon 908, chaptertimeline 910, active chapter indicator 912, chapter-event indicator 914,chapter number indicator 916, event timeline 918, chapter background920, event card 922, and timeline view icons 924. The synch button 908provides a mechanism to synch the content between the first and secondscreen devices 510, 520. The synch button 908 may also indicate thestatus of the synch between the content on the first and second screendevices 510, 520. The chapter timeline 910 indicates the chapters of theprimary content. The movie title leader is in the background of thechapter timeline 910 and indicates the primary content. As the primarycontent progresses the chapters move along chapter timeline in aconveyer-belt life fashion with the active chapter indicator 912indicating the current chapter in the primary content via highlight andcenter position of the chapter timeline 912. The chapter-event indicator914 indicates that events displayed in the event timeline 918 are partof the active chapter shown in the chapter timeline 910. The eventtimeline 918 displays event cards 922 indicating events that correspondto what is transpiring in the current chapter of the primary content.For each chapter, the first displayed event card 922 indicates thechapter that the following events occur in. As the primary contentprogresses the event cards 922 move along event timeline 918 in aconveyer-belt like fashion with the current event in the center positionof the event timeline 918. Each chapter may be provided with a uniquebackground 920 for the events of that particular chapter. The timelineview icon/button 924 indicates that the viewer is in timeline viewshowing the chapter timeline 910 and event timeline 918 as well asprovides a mechanism to access the timeline view.

The screens 900 of FIGS. 9D and 9E depict how event cards 922 progressacross the event timeline 918. Here the synch button/icon 908 indicatesthat the timeline view of the additional content is in synch with theprimary content on the first screen device 510. In FIG. 9D, the currenttriggered event card 926 is shown in the center position of the eventtimeline 918 and represents the first triggered event. To the left ofthe current triggered event card 926 in the event timeline 918 is theprevious event card 928, in this case the card indicating the chapter.To the right of the current triggered event card 926 in the eventtimeline 918 is the next event card 930, in this case the cardindicating the next scheduled event. Since, in FIG. 9D, this is thecurrent triggered event card 926 is for the first triggered event, thechapter indicator 916 indicates that it is chapter 1. The currenttriggered event card 926 includes the additional content 932 related tothe primary content. The current triggered event card 926 also providesan indicator 934 as to what type of additional content is displayed. Incertain embodiments this indicator matches an indicator shown on thefirst screen display 510. The current event card 926 also includesbuttons/icons for synching 936 and sharing 938. The synch button/icon936 provides a mechanism that causes the primary content displayed onthe first screen device 520 to be synched with the current event. Theshare button/icon 938 provides a mechanism to share the additionalcontent of the event with a social network. The elements of the screen900 of FIG. 9E are similar to the elements of FIG. 9D except that thecurrent triggered event card 926 is for an event that happens later inthe timeline as indicated by the chapter indicator 916 which indicatesthe current chapter is chapter 3.

FIG. 9F depicts examples of other possible functionality that may beprovided as part of display on the second screen device 920. Here thechapter timeline 910 is provided with a collapse icon/button 940 whichprovides a mechanism to toggle the chapter timeline between visible 940a and hidden 940 b. Likewise the synch button/icon 908 can togglebetween status indicating whether synch is currently active 908 a andstatus indicating synch has been lost and re-synching is available 908b. In some embodiments a volume button icon 942 is provided. The volumebutton/icon 942 provides a mechanism to turn the sound of the firstscreen display “OFF” or “ON”. The volume button 942 may also indicatethe status of whether the volume is “ON” indicating muting is available942 a, or “OFF” indicating sound is available 942 b. In some otherembodiments a play/pause button/icon 944 is provided. The play/pausebutton 944 provides a mechanism to pause or resume playback of contenton the first screen display 510. The pause/play button may also indicatethe status of whether the playback can be paused 944 a, or “or resumed944 b.

In the interactive approach, the user selects an additional contentevent on the second screen device 520 and what is displayed on theprimary screen device 510 is synched to the selected event. As indicatedpreviously, the events of additional content are synched to the primarycontent. If the user swipes the movie timeline or the events, the eventsbecome out of synch with the movie being shown on the main screen. Tore-synch touches the synch button on the tablet. The timeline or eventsare the synched back to what is being displayed on the main screen.Likewise, a user can select a trivia event or map event, touch the synchbutton, and the scene in the movie related to the selected trivia or mapevent will be played on the main screen. Examples of this can be seen inFIG. 10A-D.

FIG. 10A depicts how a user may interact with the chapter timeline 910and event timeline 918 on the screen 900. Here icons 1000 and 1002represent how the user can touch the screen to scroll left or right inthe chapter or event timelines 910, 918.

FIG. 10B depicts one embodiment of the screen 900 when a user interactswith the chapter timeline 910. In this example the synch button/icon 908indicates that the additional content on the second screen display 520is out of synch with the primary content on the first screen display510. Icon 1000 represents the user scrolling through the chaptertimeline 910. The current chapter remains highlighted 912 until thetransition to the new chapter is completed. When navigating through thechapter timeline 910 a chapter position indicator 1004 is provided thatindicates what chapter of the available chapters is selected. Thechapter indicator 916 also indicates the selected chapter and updateswhen the transition to the new chapter is complete. In this example,while the user is navigating through the chapter timeline 910, the eventtimeline 918 is dimmed. In certain embodiments, the user may jumpdirectly to a particular chapter by selecting the chapter from thetimeline 910.

FIG. 10C depicts one embodiment of the screen 900 when a user interactswith the event timeline 918. Icon 1002 represents the user scrollingthrough the event timeline 918. Here, the timeline 918 is beingtransitioned from current triggered event card 926 to the next eventcard 930. When navigating through the event timeline 918 an eventposition indicator 1004 is provided that indicates what event of theavailable events is selected.

FIG. 10D depicts one embodiment of the screen 900 when a userinteracting with the event timeline 918 causes a transition from onechapter to another. Icon 1002 represents the user scrolling through theevent timeline 910 causing a chapter change. Here, the timeline 918 isbeing transitioned a new event card 922 indicating a new set of eventsrelated to a new chapter. When navigating through the event timeline 918causes a transition to a new chapter the event position indicator 1004is centered until the new series of events begins.

FIG. 11A-C and 12A-B indicate some of the other interactive activitiesthat can be accessed via the event cards 922. FIGS. 11A-C depict thesocial media sharing feature. FIGS. 12A-B depict the chapter selectionas well as selection and playback of additional media files.

FIG. 11A-C shows various pop-up fields on the display 900 when thesharing feature is active via the share button/icon 937. FIG. 11A showsthe field 1100 displayed when the user has logged into their socialnetwork (in this case Facebook). Area 1102 indicates the event beingshared and area 1104 indicates the comments the user is going to shareabout the event. Button 1106 provides the mechanism to submit the eventand comments to be shared. FIG. 11B shows the field 1100 displayed whenthe user has not yet signed in to the social network. In this examplebutton 1108 is provided to sign into Facebook and button 1110 isprovided to sign into twitter. Options to sign into other socialnetworks may also be provided. FIG. 11C shows a onscreen Qwerty keyboard1112 that may be used to enter comments into area 1104 for user'scomments. In certain embodiments, this may be a default keyboardprovided by the second screen device 520.

FIG. 12A-B shows the selection of chapters as well media content forplayback by the user. In the example of 12A, if the user single taps1200 the currently playing chapter shown in the chapter timeline 912 theplayback on the first screen device 510 is paused. If the user doubletaps 1202 the currently playing chapter shown in the chapter timeline,playback of on the first screen device will jump to the beginning of thechapter and the events timeline 918 will be set to the first event ofthat chapter. In some embodiments, the event cards 922 may include mediafiles 1204 such as video or audio clips. If the media file is an audioclip, then selection of the audio clip results in playback on thecurrent screen 900. If the media file is a video clip, then selection ofthe video clip results in the launching of a full-screen media player1206 as seen in FIG. 12B. In this example the media player includeson-screen controls 1208. To return to the previous screen, the user onlyneeds to tap the non-video surface 1210 of the media player.

FIG. 13A-E depicts some other possible features regarding the additionalcontent. These include a map view 1300, family tree 1310, and settings1320. FIG. 13A depicts the menu bars for these options. In this exampleeach of these menu bars are provided with first screen device controls1330 including pause/resume and mute/un-mute. FIG. 13B depicts the mapview display 1300. The map view display 1300 includes a map 1302including marked locations 1304 and information about the locations1306. Icons are also provided to select other maps 1308. FIG. 13Cdepicts the family tree view 1310. The family tree view shows the familytree with fields 1312 indicating the relationship between the familymembers. In this example the button/icon 1314 at the bottom indicateswhat view is currently being shown (i.e. the family tree view). If afield 1312 is selected, a pop-up field 1316 is displayed, as shown inFIG. 13D, providing information about the person in the field 1312. FIG.13 e depicts the settings view 1320. In view 1320 the user is providedwith controls for adjusting the preferences for the audio and video1322, events 1324, and social network sharing 1326.

FIGS. 14A-L depict skinned examples of what may be displayed on thescreen 900 of the second screen device to a user when using anapplication that provides additional content on the second screen device520 that is synched with the primary content on the first screen device510. FIG. 14A is a skinned version of the splash screen as shown anddescribed in relation to FIG. 9A. FIGS. 14B-F depict skinned versions ofthe timeline view as seen and described in relation to FIGS. 9C-F and10A-D. FIG. 14G depicts a skinned version of a screen display whereinall the available video clips that are part of the additional contentare displayed for the user. FIG. 14H depicts a skinned version of ascreen display wherein all the available audio clips that are part ofthe additional content are displayed for the user. FIG. 14I depicts askinned version of the maps view as shown and described in relation toFIG. 13B. FIGS. 14J and 14K depict skinned version of the family treeview as shown and described in relation to FIGS. 13C and 13Drespectively. FIG. 14L depicts a skinned version of the settings view asshown and described in relation to FIG. 13E.

The events and features shown in the figures are just some examples ofpossible events. In certain embodiments, a user may be able to configureor otherwise select what events they wish to be shown (e.g., don't showme purchase events). In other embodiments the user may be able to selector bookmark events for viewing at a later time. In still otherembodiments certain events may unavailable or locked out depending onthe version of the program being viewed (i.e. purchased vs. rented or BDvs. VOD vs. Broadcast). In other embodiments, the events available canbe personalized for a user based on previous viewing habits (i.e. insystem such as TIVO where a user's viewing habits are tracked or usingthe personalization engine 118 of FIG. 1).

Other possible configurations include shopping features. For example, astore front could be provided and accessible from the second screen tofor purchasing movie merchandise. In another embodiment points or awardscould be provided to a user for watching, reviewing, or recommending aprogram or film. For example, the more movies watched or shared withfriends, the more points awarded. The points can then be used for prizesor discounts on related goods.

Similarly, achievements can also be awarded. These achievements could bepushed to a social networking site. Example achievements could include:

-   -   Watching certain scenes—Achievement    -   Watching certain discs in a series—Achievement    -   Watching certain discs by a particular studio or        actor—Achievement

In still other implementations a Wild feature could be implemented. Arunning Wiki could let a user and other users of a disc comment oncertain scene. For example, tracking metadata could be created which ispushed to a web based wild. Such metadata could include:

Chapter Information

Time Codes

Thumbnails of Scenes

Actor/Director Information

This pushed information can be used to create a running Wild which letsothers comment on the movie. These comments could then be reintegratedinto the second screen application as events which can be accessed.

Additional features and screens are also possible. For example, in someembodiments activity on one or more social networks that is related tocontent displayed on the first screen can be monitored and used toprovide additional content on the second screen.

As discussed in regard to FIG. 11A-C. The application on the secondscreen device 520 can support social media such as Facebook and Twitter.Additional examples of this can be seen in FIGS. 15-17.

FIG. 15 depict a wireframe screenshot 1500 showing the panel 1510displayed over the background, in this case a “greyed-out” or “dimmed”timeline view 1520, when the user has not yet signed in to the socialnetwork. In this example panel 1510 is for signing into twitter. Thepanel 1510 provides an area 1512 to provide a username or email addressas well as an area 1512 to provide a password. Button 1516 authorizedthe application to access twitter. Button 1518 declines the sign-in. Asimilar panel or field can be provided to sign into Facebook, or othersocial media networks.

FIG. 16 depict a flow diagram 1600 of the screens displayed to a userbased on whether they are signed into a social network. At junction 160it is determined is the user it signed in or otherwise authorized on thesocial network. If the user has not provided authorization (1612) screen1500 with panel 1510 of FIG. 15 is displayed to the user prompting themto sign in. If the user has previously provided authorization (1614) orsigns in using screen 1500 (1618) screen 1620 with message panel 1622 isdisplayed. From the message panel 1622 the user may cancel or send amessage using buttons 1624 and 1626 respectively. The text of themessage can be entered using an onscreen keyboard 1628. Once a messageis sent or canceled out of (1630) or if authorization is never provided(1632) junction 1634 is arrived at wherein the panels are dismissed andthe user is returned to the previous screen view (1640).

FIG. 17 depicts a skinned version screen 1700 of the screen 1620 of FIG.16. A message panel 1710 is provided overtop the skinned background1720. The message panel 1710 includes a text area 1712 as well as cancel1714 and send 1716 buttons. Text can be entered into the text area 1712using an onscreen keyboard 1730.

In addition to providing the ability to post social messages from theapplication on the second screen device 520, the user's experience canfurther be enhance being able to track comments on social media relatingto content being viewed on the first screen device 510 and providingadditional content on the second screen device 520 that is synched tothe primary content on the first screen device 510 based on the trackedcomments relating to the primary content.

FIG. 18 shows a flow diagram 1800 depicting one possible methodology forproviding such functionality on a second screen device 520. At a basiclevel, the method includes three steps. First, social media activityregarding content being displayed on the first screen device 510 ismonitored (step 1810). The monitored social media activity is thenprocessed (step 1820). Finally, additional content based on the socialmedia activity is provided on the second screen device 520 which issynched to the primary content on the first screen device 510 (step1830). Each of these steps will be discussed in more detail below inreference to specific exemplary implementation that rely on social mediaactivity.

Example: Social Quote Event

Popular broadcast media can potentially generate an overwhelming amountof related social messages. Second screen applications designed tosupport particular broadcasts have been developed to extend the brandedexperience. These second screen applications may include a relevantsocial feed by filtering messages by a hashtag or other keyword. Evenwith this filtering in place the message count can quickly becomeunwieldy for popular events (e.g. Superbowl 46 where the Tweets persecond peaked at 10,245). What is disclosed herein is a mechanism toidentify and surface relevant social messages for the user to seewithout being inundated with large numbers of messages to scan.

Current practice in many social messaging applications is to simplypresent the messages as they are received resulting in an almostcontinuous scrolling of messages as they are rendered in the userinterface. Other implementations may throttle requests for messages toreduce load on the back-end servers but the number of results for eachrequest may be large and would be difficult for a user to scan the listbefore the next request is fulfilled.

This disclosure offers an approach that curates social messages that arethen offered to the user with timing appropriate to a second screenapplication that may not be entirely focused on the social messagingaspect. This is valuable when it is desirable for second-screenapplications that want to integrate social messaging with other aspectsof the media experience like bonus content, trivia and advertising.

In the context of an application on a second-screen device 520 wheremultiple items are displayed to the user in a timed fashion in sync withthe media being displayed on a first screen device 510, curated socialinformation can be interleaved with other “timeline events”. For examplean actor trivia “timeline event” may be presented in conjunction with acharacter's appearance on screen. A “Social Quote Event” may be timedfor display just after an intense action scene. This latter eventdisplays only the high-level/relevant social messages based on targetingor frequency. This provides the user a sense of what is beingcommunicated in the social network while not requiring the user to scanthrough hundreds or even thousands of messages. An example of this canbe seen in FIG. 19.

FIG. 19 depicts a wireframe screenshot 1900 showing a “Social Quote”event panel 1910 as part of the timeline of other events 1920. The panel1910 provides a text area 1912 displaying the relevant curated socialquotes. In certain embodiments one or more additional buttons areprovided. In this example Button 1914 provides the ability to post or inthis case re-tweet a given quote. Other possible features andimplementation will be apparent to one skilled in the art given thebenefit of this disclosure.

The process of providing, on a second screen device 520, social mediacomments relevant to content being displayed on a first screen device510 follows the general methodology set forth in FIG. 18.

First social media activity is monitored (step 1810). This involveslooking for keywords, hashtags, or the like that include the name of thecontent being displayed on the first screen device 510, the name of theactors, the name of the director, or other related information usingtechniques and methodologies that are well known in the art. Monitoringof social media can be performed on the second screen device 520, aserver 530, provided by a third party service, or a combination ofthereof.

The step of processing (step 1820) is where the data is curated to onlyprovide the most relevant comments or messages. To achieve the curated“Social Quote” the following heuristics may be applied:

-   -   If the social message is targeted to the current user it should        be deemed important for display. As an example this is like an        “at” message (@user) in Twitter. Other targeting or addressing        schemes can be employed as well (e.g. post on a wall in        Facebook)    -   Many social messages are simply re-sending original messages.        Twitter provides a mechanism of “re-tweets” that adds metadata        to a message to identify how many times it has been passed        along. In this case the re-tweet count provides a metric of        “importance” which is then used to select these messages for        display.    -   if the message contains multiple keywords it can be given        greater “importance”.    -   if a user may have specified user preferences for which the        messages can be compared to.

In certain embodiments is also conceivable that social messages can be“sponsored” so that they automatically gain relative importance abovethe crowd of other messages. For example social media comments that wereposted from a second screen application may be given priority. A furtherrefinement to this would be along the lines of targeted advertising—theimportance of sponsored messages can be weighted against user-specificcriteria to achieve its relative ranking amongst other messages. Thisprocessing or curating can be performed on the second screen device 520,a server 530, provided by a third party service, or a combination ofthereof.

One or a few social messages thus ranked can be displayed to the user toprovide targeted and other “significant” messaging without having toresort to scanning ever-flowing lists of social messages (step 1830).

In the case of the timeline-based application on a second screen device520, “Social Quote” events can be distributed throughout the timelinesuch that regular exposure to social networks is purposefullyinterleaved with other event types for a thoughtfully designedexperience.

Example: Dynamic Advertising

A key to effective advertising is to deliver relevant offers at theright time. Current techniques involves extensive data collection onuser behavior (e.g. Google tracking your searches, application use,viewing habits, etc.) which is then used to select advertisements thatbest suit the user's profile. A great deal of specificity can be derivedusing public and private data to develop these personalization profiles.

Providing a focused context for collecting data has the potential forimproving advertisement delivery even further. Social messaging mayindeed be used as a source for personalization but may be handledindependently from other activities that the user may be engaged insimultaneously. For example a user may tweet about a particular movieand a back-end system in the cloud can record the interest in thatmovie. This data could then be used the next time a user visits awebsite that takes advantage of this data to deliver advertisements.While this may potentially result in a higher advertisement relevance tothe user during the subsequent visit the context of the original socialmessage is lost. Second-screen applications offer a known environmentfrom which social messages can be monitored to enhance advertisingpersonalization.

An application on a second screen device 520 can designed to displayscheduled “events” in sync with primary video playback. Events canrepresent trivia, social content, voting, bonus material,advertisements, etc that are timed for display at relevant points in theprimary video playback.

The context that the second-screen application (and device) provides isrich, to include:

-   -   Media information. This establishes an interest in a particular        media item (broadcast TV, Video, etc)    -   Specific Media time information. The application knows where you        are within a piece of media and any specific metadata associate        with that time.    -   Location information. Second-screen devices 520 are typically        equipment with mechanisms for identifying your current location

The disclosed embodiments interleave “events” to support an overallexperience. Some of these event types afford a social interactionelement such as voting or can offer social messaging anytime throughoutthe presentation. Subsequent “events” can be reserved for advertisingand can react to previous activity to include social messaging. This canbe accomplished by identifying keywords within the social message itselfand providing that to an advertising service along with other relevantinformation such as location, metadata associated with the current mediatime, etc.

The application consolidates all these variables and offersadvertisements within the context of the overall experience.

For Example a James Bond film could be playing on a first screen deviceand a location such as Hong Kong is displayed on screen. Thesecond-screen app displays a supporting media item such as trivia aboutthe location. The user could send a social message (tweets) to describeher desire to go on a vacation. The word “vacation” is parsed by theapplication in preparation for later advertising display. Later in thesecond-screen timeline an advertisement “event” focusing on a vacationoffer is displayed.

This process also follows the general methodology 1800 of FIG. 18. Themonitoring (step 1810) takes place when it is detected that the user hassent a social message including the keyword “vacation.” In this example,the monitoring is performed on the second screen device 520, but iscould also be performed at the server 530, by a third party service, ora combination thereof.

In the processing step (step 1820), the specific advertisement isselected from an ad service by submitting the keyword “vacation”.Additional information such as the specifically known location in themovie can be derived from metadata associate to the movie at theparticular time of the tweet. Additionally the advertisement canrecommend a local travel agent using the user's location information.The processing can be performed on the second screen device 520, but iscould also be performed at the server 530, by a third party service, ora combination thereof.

Finally, the advertisement can be displayed on the second screen device(Step 1830). An example of this can be seen in FIG. 20.

FIG. 20 depicts a wireframe screenshot 2000 showing a dynamicadvertising event panel 2010 as part of the timeline of other events2020. The panel 2010 provides a text area 2012 displaying the relevantadvertising offers. Other possible features and implementation will beapparent to one skilled in the art given the benefit of this disclosure.

The advertisement thus becomes a natural part of the “conversation” allwithin the context of the media consumption experience.

While the addition of social messaging to a second screen application asdiscussed in the examples above enhances the experience for the user,further functionality and enhancement of the experience can be achievedif the social messaging can be time-stamped or otherwise synched withthe playback of content on the first screen device 510.

Example: Synching Social Messages

Social messages are typically ephemeral and relevant only to the presentmoment in time. A common convention for associating a message to aparticular event or topic is to use hashtags. These hashtags provide ameans to filter for social messages of interest. Filtering for thesehashtags provides a “real-time” view of messages on the topic. Thisdisclosure takes this notion a step further and describes a mechanismfor associating message to a specific point in time relative to thestart of a media content (such as video). Social messages employingthese techniques can be associated to specific points in time within theparticular content. For example, social messages can be associated withan opening scene that sparks social commentary and later for otherarbitrary points in the media timeline. This example becomes even morerelevant when a piece of media is replayed at a later time.

In one embodiment, hashtags are used to add a timestamp. The currentpractice of using hashtags is well known. There is currently no widelyestablished mechanism for describing time offset information within asocial message or as additional metadata. The present disclosureprovides a mechanism that can be implemented without custom extensionsto the social messaging protocol. The encoded time offset is sent aspart of the message itself. It is possible to obviate this approach byproviding specific metadata that is not typically displayed as part ofthe social message itself.

FIG. 21 depicts exemplary social messages 2100 that including a hashtagidentifying the media 2110 is provided (this is common use today) withthe addition of another hashtagged string of characters encoding thetime offset 2120. In this example, the hashtag #ijkcs (2110) is used toidentify Indiana Jones and the Kingdom of the Crystal Skull. The hashtag#1054675(2120) is a checksummed encoding of the frame offset.

FIG. 22 depicts an exemplary flowchart of a method for creatingtime-stamped social messages. The method begins at block 2210. It isthen determined that a user desires to create a social message (block2220). In this embodiment, this occurs when the user selects a commentbutton provided as part of the application running on the second screendevice 520. A timestamp of the position of the playback is then created(block 2230). In this example, this involves generating a checksum. Tocreate the checksum the application receives positioning informationdirectly if the media is being played back within the application orfrom other methods that provide position information from externalsources. This numeric data is then checksummed. The user can then enterthe text for the social message and request it be sent (block 2240). Thetimestamp is then added to the message and the message is sent orotherwise committed to the social network (block 2250). In this example,this involves appending the checksum to the social message text. Thechecksum is used to ensure integrity of the data and allow a consumingapplication to ignore bad position data that might be maliciously orinadvertently created. This ends the method (block 2260)

FIG. 23 depicts an exemplary flowchart of a method for consuming orotherwise reading and decoding a time-stamped social message. Once begun(block 2310) the method includes receiving a social message pertainingto the particular content in question, in this case, the content beingdisplayed on the first screen device 510 (block 2320). In this examplethis involves the selection of a message based on the hashtagrepresenting the video of interest (basic search on hashtag). Theposition information (timestamp) can then extracted (block 2330). Inthis example, this involves the application looking for an additionalhashtag that immediately follows and processed as an encoded position.If the position information is found (block 2340), The positioninformation is separated into a predetermined position and checksum. Theposition is confirmed against the checksum (block 2350). If the positioninformation is valid, the position can then be used to associate thesocial message to a particular time within the video and the messageinformation can be displayed synched with the display on the firstscreen device 510 (block 2360). If the position information cannot beconfirmed, the message can be discarded or displayed without beingsynched with the display on the first screen device 510 (block 2370).This ends the method (block 2380).

Assuming not all social commentary will be generated using anapplication employing the techniques of the present disclosure, thefollowing approach may be used to modify message so they can be consumedby applications sensitive to the position information.

A separate service (e.g. in the cloud) can monitor and record real-timefeeds of social messages of interest (say for a particular televisionbroadcast). This service could use known broadcast schedules andcorrelate the expected time of the television event with the time themessages are seen in real-time. The service then records the socialmessage with the appended position information. The resulting repositoryof time-stamped message is then accessed in lieu of a direct connectionto the social network. In certain embodiment origin information for thesocial messages can be used to correlate to specific regions withspecific broadcast times.

Example: Heat Map

As discuss previously, social messaging is becoming a ubiquitous featureacross various software applications. One problem this presents is thatthe sheer quantity of messages can become overwhelming especially forpopular topics. At some point the individual messages get lost in thecrowd and the quantity of messages becomes the interesting socialaspect. This disclosure describes a way to visualize the volume ofsocial activity over time which can then be used to identify interestingpoints in time for media content and also be used to navigate to thosepoints in time.

Many second-screen applications provide a social message “feed” thatsimply streams social messages as they happen. There is littleorganization of this information beyond displaying the most recent atthe top of the list or perhaps the notion of a “promoted” message thatadvertisers use to keep their message at the top of the stack. Messagesare quickly replaced by new messages sometimes faster than a user canscan them. Once the messages have been buried in the stack theireffective relevance to time is diminished.

The presented graphical representation referred to here as a SocialHeatmap organizes social information to correspond with the timeline ofa of a media item such as content being displayed on a first screendevice 510. In the case of live broadcast the social message timestampcan be used to make the correlation. In the case of recorded content,messages need to be stamped with the time relative to the start of themedia. This can be done within the application since the relative timeof the media playback can be acquired via syncing mechanisms. The synctime is added to the social message which is then made available via asocial network to others using an application that is aware of the timeinformation. In this way each social message has a relative mediatimestamp which aids in visual placement on the screen when displayingthe information on a second screen device 520.

FIG. 24 depicts an exemplary flowchart of a method for consuming orotherwise reading and decoding a time-stamped social message. Once begun(block 2410) the method includes receiving a social message pertainingto the particular content in question, in this case, the content beingdisplayed on the first screen device 510 (block 2440). This can involvethe selection of a message based on the hashtag representing the videoof interest (basic search on hashtag) as discussed above or it could bebased on keywords in the messages themselves. Timestamp informationassociated with the message is then looked for and messages withoutassociated timestamp information are discarded (block 2430). In certainembodiments this involves the application looking for an additionalhashtag that immediately follows and processed as an encoded position.Based on the associated timestamp information, the messages can beallocated to bins (block 2450). This process is discussed in more detailbelow. A graphical representation of the social messages associated withspecific time periods of the primary content on the first screen device510 can then be displayed on the second screen device 520 while theprimary content is being displayed on the first screen device 510. Incertain embodiments the graphical representation of the social messagesassociated with specific time periods also graphically represent theintensity or frequency (i.e. the number) of social messages associatedthe specific periods of time and as such is referred to as a heatmap.The user can then use the heatmap to navigate the through the content onthe first screen and the associated social messages (block 2460). Suchnavigation is discussed in more detail below. This ends the method(block 2370).

Bins are mechanism used for grouping messages with discrete sections oftime. The number of messages in each bin can corresponds to the activitylevel for that section of time. It has been found that the screen widthof each bin should be wide enough to afford navigation (e.g. using atouch device) but small enough to provide navigation resolution. Anexample of this can be seen in FIGS. 25 and 26.

In the example of FIG. 25, a tablet application allocates 1000 pixels ofwidth for the heatmap 2500. These 1000 pixels represent the entirety ofthe media playback time, say 50 minutes. This allows approximately 20pixels of screen width per minute of content. If we use a bin width of20 pixels then our navigation resolution will be to the nearest minuteand the sensitivity for navigation will require a move of 20 pixels frombin to bin. FIG. 26 depicts an exemplary plot of sample count for valuesallocated to bins for rendering. The number of messages in a bin may begraphically represented using color, plots, or other indicators. Theactual sensitivity that should be used will depend on the input device,user demographics and other factors.

FIG. 27 depicts an exemplary wireframe screenshot 2700 including asocial heatmap or timeline. In this embodiment event panels 2710displayed as part of a timeline view on the second screen device 520which a user can scroll through as described previously. A Commentbutton 2720 is provided to allow the user to send social messagesregarding content being displayed on the first 510 or second 520 screendevices. Social messages generated using such a function can includecontent and timestamp information for processing and inclusion in theheatmap 2750. Sync indicator button 2730 allows the user to synchronizethe events of the timeline view with the content being displayed on thefirst screen device 510. Button 2730 can also indicate the status of thesynchronization. At the bottom of the screenshot 2700 there is also aplayback position indicator 2740 and the social heatmap 2750. Theplayback position indicator graphically displayed the current positionin the playback of the primary content on the first screen device 510.In certain embodiments, a user can adjust the indicator to change thecurrent playback position in the primary content being displayed on thefirst screen device 510. The social heatmap 2750 has bins representingsocial messages associated with time periods in the playback of theprimary content. Selecting bins, for example by sliding or “scrubbing”along the social heatmap 2750 causes the associated social messages tobe displayed. Examples of this can be seen in FIGS. 28-32.

FIG. 28 depicts the screenshot 2700 of FIG. 27 with a pop-over panel2800 that is displays the social messages associated with a bin 2810 inthe social heatmap 2750 when a specific bin is selected by a user asrepresented by icon 2820. FIG. 29 depicts one embodiment of how messagescan graphically be transition between (as indicated by arrow 2900) inthe popover panel as a user scrolls or scrubs along the heatmap 2750 (asindicated by icon 2820). FIG. 30 indicates one possible embodiment of apanel 3000 that can be displayed if there are no social messagesassociated with a bin 2810. FIG. 32 depict how multiple messages can bescrolled through within a panel 3100. In this example, messages can bescrolled through vertically as indicated by arrows 3110. In certainembodiments a scroll bar indicator 3120 is provided to indicate thatthere are multiple messages to be scrolled through. FIG. 32 depictanother embodiment wherein selecting individual messages within thepanel 3200 provides the user with additional functionality. In thisexample, selecting a message provides additional buttons that allow theuser to resend (“re-tweet”) a message 3210 or got the specific instancein the playback of the content on the first screen device 510 for withthe message is associated.

FIG. 33 depicts a skinned version screen 3300 of the screen 2700 of FIG.27. In this example color and peaks are used to graphically indicate theintensity of social messages along the heatmap 3310.

The present description illustrates the principles of the presentdisclosure. It will thus be appreciated that those skilled in the artwill be able to devise various arrangements that, although notexplicitly described or shown herein, embody the principles of thedisclosure and are included within its spirit and scope.

All examples and conditional language recited herein are intended forinformational purposes to aid the reader in understanding the principlesof the disclosure and the concepts contributed by the inventor tofurthering the art, and are to be construed as being without limitationto such specifically recited examples and conditions.

Moreover, all statements herein reciting principles, aspects, andembodiments of the disclosure, as well as specific examples thereof, areintended to encompass both structural and functional equivalentsthereof. Additionally, it is intended that such equivalents include bothcurrently known equivalents as well as equivalents developed in thefuture, i.e., any elements developed that perform the same function,regardless of structure.

Thus, for example, it will be appreciated by those skilled in the artthat the block diagrams presented herewith represent conceptual views ofillustrative circuitry embodying the principles of the disclosure.Similarly, it will be appreciated that any flow charts, flow diagrams,state transition diagrams, pseudocode, and the like represent variousprocesses which may be substantially represented in computer readablemedia and so executed by a computer or processor, whether or not suchcomputer or processor is explicitly shown.

The functions of the various elements shown in the figures may beprovided through the use of dedicated hardware as well as hardwarecapable of executing software in association with appropriate software.When provided by a processor, the functions may be provided by a singlededicated processor, by a single shared processor, or by a plurality ofindividual processors, some of which may be shared. Moreover, explicituse of the term “processor” or “controller” should not be construed torefer exclusively to hardware capable of executing software, and mayimplicitly include, without limitation, digital signal processor (“DSP”)hardware, read only memory (“ROM”) for storing software, random accessmemory (“RAM”), and nonvolatile storage.

Other hardware, conventional and/or custom, may also be included.Similarly, any switches shown in the figures are conceptual only. Theirfunction may be carried out through the operation of program logic,through dedicated logic, through the interaction of program control anddedicated logic, or even manually, the particular technique beingselectable by the implementer as more specifically understood from thecontext.

Although embodiments which incorporate the teachings of the presentdisclosure have been shown and described in detail herein, those skilledin the art can readily devise many other varied embodiments that stillincorporate these teachings. Having described preferred embodiments fora method and system for providing media recommendations (which areintended to be illustrative and not limiting), it is noted thatmodifications and variations can be made by persons skilled in the artin light of the above teachings.

While the example set forth above has focused on an electronic device,it should be understood that the present invention can also be embeddedin a computer program product, which comprises all the features enablingthe implementation of the methods described herein, and which, whenloaded in a computer system, is able to carry out these methods.Computer program or application in the present context means anyexpression, in any language, code or notation, of a set of instructionsintended to cause a system having an information processing capabilityto perform a particular function either directly or after either or bothof the following a) conversion to another language, code or notation; b)reproduction in a different material form.

Additionally, the description above is intended by way of example onlyand is not intended to limit the present invention in any way, except asset forth in the following claims.

What is claimed is:
 1. A method for synching social messages to acontent timeline, the method comprising: monitoring social media for amessage relating to displayed content; processing the message toassociate the messages with a period of time in the display of thecontent; and associating a timestamp with the message indicating theperiod of time in the display of the content for which the message isassociated.
 2. The method of claim 1, wherein the social media beingmonitored is Twitter.
 3. The method of claim 2, wherein the timestampcomprises a hashtag appended to the social message.
 4. The method ofclaim 3, wherein the timestamp further includes a checksum.
 5. Themethod of claim 1, wherein the displayed content is being displayed on afirst screen.
 6. The method of claim 1, wherein the method is performedon a second screen device.
 7. The method of claim 1, wherein the methodis performed on a server.
 8. The method of claim 1, wherein the step ofprocessing is performed based in part on the location of origin forsocial message.
 9. A method for processing social media synched with acontent timeline: receiving a message relating to displayed content;determining if a timestamp is associated with message indicating theperiod of time in the display of the content for which the message isassociated; and providing the message synched to the display of thecontent.
 10. The method of claim 9, wherein the social media beingmonitored is Twitter.
 11. The method of claim 10, wherein the timestampcomprises a hashtag appended to the social message.
 12. The method ofclaim 11, wherein the timestamp further includes a checksum.
 13. Themethod of claim 9, wherein the displayed content is being displayed on afirst screen.
 14. The method of claim 9, wherein the method is performedon a second screen device.
 15. A second screen device comprising: ascreen configured to display content; storage for storing data, and aprocessor configured to receive a message relating to displayed content,determine if a timestamp is associated with message indicating theperiod of time in the display of the content for which the message isassociated, and provide the message synched to the display of thecontent.